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Feasiblity of an On-line Concept Mapping Construction and Scoring System 

Purpose 

The purpose of this presentation is to provide an overview of and assess 
the feasibility of the administration, scoring, and reporting of on-line concept 
mapping systems. In our collective studies, computer-based concept mapping 
assessments were designed to measure students' environmental science 
content knowledge by requiring them to construct semantic relationships 
among important concepts in the content domain. Some students were also 
asked to search a simulated Internet database for information about these 
important concepts, so as to measure problem solving. Concept map 
construction systems were designed to allow students to individually and 
collaboratively (in groups of three) construct concept maps on the computer, so 
as to measure both content understanding and collaboration. 

Theoretical Framework 

The motivating concern behind our study has been the desire to assess 
students' 1 understanding of relationships utilizing a format that departs from 
both standardized testing and discourse-dependent tasks (e.g., essays). In our 
research, conceptual knowledge is represented in the form of concept mapping, 
(Baker & Niemi, 1991; Baker et al., 1994; Baker et al., 1990; Baker, Niemi, et 
al., 1992; Dansereau & Holley, 1982; Herl, Baker, & Niemi, 1996; Holley & 
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Dansereau, 1984; Lambiotte, Dansereau, Cross, & Reynolds. 1989) 
comprising nodes and links, nodes representing concepts or their attributes, 
and links expressing the semantic relations among those concepts. Empirical 
research on the efforts of concept mapping (Collins, Dansereau, Holley, 
Garland, & McDonald 1981; Dansereau et al., 1979; Holley, Dansereau, 
McDonald, Garland, & Collins, 1979; Lambiotte & Dansereau, 1991; Rewey, 
Dansereau, Skaggs, Hall, & Pitre, 1989; Roth & Roychoudhury, 1992) found 
that students using concept mapping strategies performed significantly 
better on text processing tasks than students using their own methods. The 
value of mapping as an instructional tool implies the desirability of having 
methods for constructing and scoring concept maps as a way to link 
assessment and instruction. 

Concept map construction and scoring systems. A concept mapping 
construction system contains terms, links, and map construction procedures. 

A term can represent more tangible things, such as people, places, and things, 
or less tangible things, such as ideas. Links are used to represent semantic 
relationships among concepts. Specifications for map construction systems 
should provide the necessary procedures for users for construct those semantic 
relationships. The importance of the specifications of the system is 
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Feasiblity of an On-line Concept Mapping Construction and Scoring System 

independent of the testing format (i.e., on-line or paper -pencil task) used to 
administer the assessment. 

There are two main classifications of map construction systems: 

1) open, and 2) closed. An open construction system is defined as one which 
requires map constructors to provide their own terms and links. A closed 
system contains finite sets of terms and links, from which constructors must 
choose in order to construct their maps, and provides the model for the map 
construction system used for this study. 

Concept maps have typically been constructed by individuals using 
paper-pencil formats, in which the map constructor draws the concept map by 
hand. Currently, there are two problems with the implementation of paper- 
pencil concept mapping construction systems: (1) the cost of transcribing 
student-constructed maps from paper-pencil format to computerized data 
format is directly proportional to the number of students and is not yet 
feasible for large-scale assessment, and (2) the percentage of students’ 
missing data might be considered unacceptable to some. Herl et al. (1996) 
collected over 300 paper-pencil concept maps, and reported that more than 4% 
of students had to be eliminated from data analyses because their maps 
contained less than a minimum percentage of valid links (in that study, 80% 
was considered acceptable ) in their concept maps. 
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Feasiblitv of an On-line Concept Mapping Construction and Scoring System 

In summary, map construction systems based on hierarchical memory 
models must have scoring systems which are based on criteria reflecting a 
hierarchical nature of cognitive structure, while no such restrictions on 
associationist memory-based mapping systems. The selection of memory 
model and the use of expert performance in the scoring system are 
independent of the decision to use open or closed construction systems. When 
open map construction systems are implemented, however, their respective 
scoring systems must include human raters. Such human-based scoring 
systems are expensive and prone to error. Our study attempted to implement 
a closed map construction system using expert-constructed concept maps to 
provide automated scoring and feedback for student-constructed concept 
maps. 



Methods 



Participants 

Four middle school and high school teachers’ classes were recruited to 
participate in a study about environmental science. Overall, 138 middle 
school and high school students were randomly selected from three high 
schools and one middle school, yielding a total of nine classes. Class sizes 
ranging from 17 to 28. English was the primary language for all students. 
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Students were randomly selected in groups of three to participate in 
collaborative concept mapping tasks or individually to complete problem- 
solving and search tasks. There were 69 students selected for the group 
mapping task and 69 students for individual problem-solving and search 
task. 

Equipment and Setup 

Each classroom contained at least four Macintosh 8500s, each one 
containing 24MB RAM and 1GB hard disk storage. An ethernet network was 
configured with two 10T 8-port hubs, in-classroom MACs, and five additional 
Power Mac 5300/16MB/1GB portable computers. Both the Macintosh 8500s 
and 5300s had 17” multiscan color monitors connected to them. 

Measures 

Each student completed one of two on-line concept mapping tasks. One 
of the tasks was a collaborative task and the other was defined as the 
problem-solving and search task. 

Collaborative concept mapping task. The collaborative concept 
mapping task is designed to measure students’ content knowledge and 
collaboration skills by requiring them to construct semantic relationships 
among important concepts in particular content domains. Students were 
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assigned to three-member teams and asked to collaborate on constructing 
computer-based concept maps. This task required each student to work on 
his/her own computer and collaborate with other team members using 
CRESST-supplied HyperCard® based concept mapping software. Figure 1 
displays an example of a concept map constructed using the collaborative 



mapping system. 
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Feasiblity of an On-line Concept Mapping Construction and Scoring System 

A three -member group consisted of two members and a leader. The 
concept mapping software randomly selects one member from each group to 
serve as the first leader; however, each member of a group has an opportunity 
to assume this pivotal role twice during the task. The leader is the only team 
member to have the ability to make changes to the group concept map; the 
other two members only have visual access to the map. Any additions or 
revisions made to the map by the leader are automatically updated on the 
screens of all group members. On-line collaboration between group members 
takes place using a pre-defined messaging system. In this data collection 
effort a pre-defined message system containing 35 different messages was 
implemented. Each of the three group members served two six-minute terms 
as leader of the group during the 36-minute task. 

The first step in constructing a concept map was to select a term from 
the pull-down menu and drag it into place anywhere on the screen. 
Subsequent steps included selecting other terms from the menu and 
constructing semantic relationships among pairs of concepts. In this study, 
sets of environmental science concepts (atmosphere, bacteria, carbon dioxide, 
CLIMATE, CONSUMER, DECOMPOSITION, EVAPORATION, FOOD CHAIN, GREENHOUSE 
GASES, NUTRIENTS, OCEANS, OXYGEN, PHOTOSYNTHESIS, PRODUCER, RESPIRATION, 
SUNLIGHT, WASTE, WATER CYCLE), link labels (CAUSES, INFLUENCES, PART OF, 
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produces, requires, used FOR. USES), and pre-defined collaborative messages 
were provided to each student of the group. 

Problem-solving and search task. The problem-solving task is also a 
computer-based task containing two distinct activities. The first activity of 
the task required students to individually construct concept maps, which 
serve as pretest measures of content understanding. For this task, 
environmental science content was assessed for both tasks, and therefore the 
same sets of concepts and links were used in the collaborative mapping and 
problem solving and search tasks. Students were allowed 20 minutes to 
complete their individual maps, after which they were provided with general 
feedback concerning which concepts needed the most improvement. This 
feedback was designed to be general in the sense that information was not 
provided to students specific to their incomplete content understanding 

The second part of the task required students to search through a 
CRESST-developed simulated web space containing information about the 
set of environmental science concepts. Students were provided with the 
ability to search the simulated web space via a: 1) searching interface, 

2) directory, or 3) glossary. Students were instructed to bookmark any web 
pages they found to be specifically relevant to particular concepts. They were 
also instructed to add to or revise their concept maps based upon the 



12 

o 

ERLC 



10 



Feasiblity of an On-line Concept Mapping Construction and Scoring System 



information they found resulting from their searches. An additional 40 
minutes were provided to students to complete the second activity of the 
problem-solving and search task. Figure 2 displays a concept map constructed 
during this task by a 9th grade science student. 
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Figure 2 

Problem Solving and Search Concent Manning Task Screen 




Collaborative concept mapping task. Students participating in the 
collaborative concept mapping task received 12 minutes of hands-on training 
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using our HyperCard® concept mapping software. One CRESST researcher 
was assigned to conduct collaborative concept mapping training sessions. 

The training sessions emphasized the skills necessary for constructing 
concept maps (adding, linking, moving, and erasing concepts), and sending 
messages to other group members. The training session was modeled after 
previous studies (Baker et al., 1990; Baker & Niemi, 1991; Herl, 1995, 1996), 
but was modified to adapt to the concept mapping construction system 
designed for this study, which comprised on-line training prompts. Part of the 
training session was designed to provide definitions for each of the links used 
in the concept mapping task. 

The demo software allowed students to practice constructing, revising, 
and erasing links, as well as adding, erasing, and moving concepts. It is 
theorized that very different sets of concepts should be used for concept 
mapping task training sessions than are used for the task itself. The training 
set used for the demo included concepts which were not related to the 
concepts used in the collaborative task, and was considered to comprise a 
more “general” knowledge content domain. Subsequently, the set of concepts 
used in the training session consisted of common terms related to birds, 
including: 1) BIRD, 2) EAGLE, 3) FLIGHT, and 4) WINGS, and the set of training 
links consisted of: 1) part OF, 2) type OF, 3) USED FOR, and 4) USES. These sets 
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were contained relatively few concepts and links, serving to provide more 
manageablity and standardization for the training sessions. Overall, 48 
minutes were allotted for the collaborative concept mapping task, including 
12 minutes for training and 36 minutes for groups of students to complete 
their collaborative concept maps. 

Problem-solving and search task. For this task, students received 

eight minutes of hands-on training using CRESST supplied JAVA® concept 

mapping software. Although there were some JAVA® graphic user-interface 
differences, the training sessions were similar to those used for the 
collaborative mapping task. Training procedures were compatible with those 
used for the collaborative task; however, the problem-solving and search task 
is completed by individuals, and does not include any collaborative 
components. After students had completed their 20 -minute concept maps, 
there was an additional 10-minute hands-on training session given to 
students. The training emphasized each of the three search interfaces 
provided (search, directory, and glossary). They were also instructed on how 
to bookmark relevant web pages found during their searches, and how to 
revisit those pages which had previously been bookmarked. A total of 78 
minutes were provided to students for the completion of the problem solving 
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and search task, including 18 minutes for training, and 60 minutes to 
complete both the individual mapping and searching activities. 

Scoring Procedures 

Collaborative concept mapping task . Four teachers considered to be 
knowledgeable in the field of environmental science were recruited to 
construct concept maps, which were subsequently used as sources of expert 
criteria to score students' maps. Scores were computed for each group to 
measure 1) content knowledge contained in the group concept map, and 2) the 
amount and type of group collaboration. 

There are two scores associated with measuring the content knowledge 
contained in concept maps. The semantic content score of a concept map is 
based on the semantic links constructed by experts in their concept maps. 
Two different methods of using expert maps to score student maps for 
semantic content were tested: (1) stringent, and (2) categorized. These two 
methods were created to compare the effects of removing subtle differences in 
the semantic meanings of the links. For each of these two methods, each 
expert's map was used to compute a total map score for each student, 
resulting in four scores for each student map. The second method 
incorporated clustering of similar kinds of links. To compute categorized map 
scores, some links were first categorized into different sets of link types. For 
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example, causes, entluences, and produces were categorized and labeled as 
causal links. The second method obviously generates map scores which are 
greater than or equal to the scores using method one. The number of links 
constructed in each expert's map represented the maximum possible score for 
each student map. Since exact agreements and reliabilities using the two 
methods were similar, categorized map scores were used in the analyses to 
serve as the semantic content scores because they contained more variance 
resulting from a greater range of map scores. 

The organizational structure score is defined as the similarity between 
students' and experts' maps. For each student map, organizational structure 
was calculated using a metric measuring the degree of similarity between 
neighborhoods of network terms (Herl et al., 1996). The range of the 
organizational structure score is 0 to 1, where 0 represents no structural 
similarity at all between the student map and the expert map; and 1 
represents a perfect structural match between the student map and the 
expert map. 

Problem-solving and search task. For each student’ s map constructed 
during the problem-solving and search task, two measures were computed 
related to the content contained in the map, and a list of the URLs for the web 
pages that were searched and bookmarked were logged. Semantic content and 
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organizational scores were computed for the pretest concept map (after the 
first 20 minute mapping activity) and posttest map (after the 40 minute 
searching activity). Each of the 252 web pages in the simulated internet 
database was scored a priori by two raters. Those raters were trained to score 
each of the web pages on its relevance to each of the concepts used in the 
mapping task (Schacter et al., 1997), and averages of the two raters’ scores 
were used to score students’ bookmarks. 

Analysis 



Descriptive statistics 

There are four descriptive statistics associated with concept maps 
presented here, including: (1) semantic content score, (2) organizational 
structure score. (3) number of terms used, and (4) number of links. The first 
two statistics were computed using expert concept maps as sources of criteria 
and the last two were calculated independent of experts’ maps. A term was 
considered to be used in a concept map when there was at least one link 
connected to it. The number of links is also a countable statistic and is 
defined as the number of links constructed. There were 22 group concept maps 
scored according to expert criteria. One group was excluded from the analysis 
because of problems with one of the computers used in the task. 

ERIC 
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Table 1 displays the descriptive statistics for the collaborative concept 
maps, problem solving and search pre- and post- maps, and experts’ concept 
maps. Students’ semantic content scores ranged from 0.75 to 15.5 and were 
slightly positively skewed, while their organizational structure scores ranged 
from 0.10 to 0.37. Experts’ semantic scores ranged from 20.7 to 25.3, while 
their organizational scores ranged from 0.41 to 0.51. It can readily be inferred 
from the reported statistics that experts’ map scores would be significantly 
higher than students on each of the first two concept mapping task measures. 
Obviously, one rationale is that they have higher content knowledge, which 
allows them to construct more correct links among the set of environmental 
science concepts. 




18 



Feasiblity of an On-line Concept Mapping Construction and Scoring System 



Table 1 

Means and Standard Deviations for Collaborative, Problem Solving and Search 
Task, and Experts’ Concept Mapping Task Measures 



Problem Solving 

Collaborative a Pretest^ 5 Posttest c Experts^ 



Measure 



Semantic content score 



M 


5.17 


3.81 


4.28 


22.50 


SD 


3.41 


2.86 


3.98 


2.15 


Organizational structure score 


M 


0.22 


0.17 


0.18 


0.46 


SD 


0.08 


0.09 


0.11 


0.04 


Number of terms used 


M 


14.68 


12.73 


13.02 


18.00 


SD 


3.75 


5.07 


5.13 


0.00 


Number of links 


M 


18.41 


13.77 


14.57 


44.75 


SD 


6.50 


8.59 


9.36 


10.31 



Note. a s =22. b n=64. c n=63. d n=4. 



The next speaker will present results and discussion concerning the 
relationships among the collaborative mapping measures and six teamwork 
processes, including: 1) adaptability — recognizing problems and responding 
appropriately, 2) communication - the exchange of clear and accurate 
information, 3) coordination — organizing team activities to complete a task 
on time, 4) decision-making - using available information to make decisions, 
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5) interpersonal - interacting cooperatively with other team members, and 6) 
leadership — providing structure and direction for the team (Chung et 
al., 1997). The Final speaker will present analysis and discussion of the 
relationships among pre- and post-test mapping measures, information 
seeking and retrieval measures (bookmarking), and search processes collected 
in the problem-solving and search task (Schacter et al., 1997). 

Agreement Results for Concept Mapping Tasks 

Exact agreement analyses were performed on the pairwise agreement 
using experts’ maps to score students’ maps. Each expert’s map was used to 
score each link in students’ concept maps. A pair of experts is considered to 
agree on the correctness or incorrectness of student’s links. There are three 
possibilities for each student-constructed link: 1) both experts construct links, 
2) only one expert constructs a link, and 3) neither expert constructs a link. A 
link was scored as incorrect when it did not match either of the expert’s links, 
or whenever both experts did not construct links. There are two possibilities of 
how pairs of experts might disagree: 1) experts disagree on whether to score a 
particular link as correct or incorrect, or 2) only one expert constructs a link 
between the same two concepts as the student. In this study, agreement is 
defined as both experts agreeing on the score of a particular link, whether or 
not both experts constructed links. The number of links in each student’s map 
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represents the total number of comparisons made for that particular map. 
Exact agreement is defined as the number of agreements divided by the total 
number of comparisons. 

The inter-expert agreement results using expert maps as criteria to 
score student maps are presented in Table 5 for all pairwise combinations of 
experts. As may be seen in Table 2, the exact agreement percentages are 
reasonably high (in the 80 percent range). However, it should be noted that if 
the more strict definition of agreement is used, that is, when experts are 
considered to disagree when only one of a pair constructs a link between the 
same concepts ( even though the student’s score would be scored 0 by both 
experts), the exact percentage of agreements would range from 75% to 82% for 
the collaborative mapping task, and 78% to 83% for the problem solving task. 
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Table 2 

Inter-Expert Pairwise Agreement Percentages on Concept Map Scores 



Problem Solving 
Collaborative a Pretest* 5 Posttest c 



Rater Pair 



2 


1 


85 


86 


88 


3 


1 


88 


91 


90 


3 


2 


84 


87 


88 


4 


1 


82 


85 


87 


4 


2 


89 


89 


90 


4 


3 


86 


89 


89 



N°t e - a n=22. * 5 n=64. c n=63. 

Discussion 



An overview of two different concept mapping construction and scoring 
systems has been discussed in this paper. Preliminary student achievement 
results have been presented for two concept mapping measures derived from 
expert-based map scoring systems. The pairwise agreement results for those 
scoring systems appear to be very good, however, very few studies have 
reported these kinds of results for concept mapping tasks. In fact, very few 
studies have implemented map scoring systems based on expert performance 
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as sources of criteria. The next speaker. Greg Chung, will now present a more 
detailed discussion of the results of the collaborative mapping task. 
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